Historical and Recent Trends in the Forecasting Literature
Talk for the WARN-D Friday Science Meeting
February 16, 2024
Introduction
Why talk about this topic?
the secret reason:
Ok but really, why should you care?
“The quiet revolution of numerical weather prediction” (Bauer, Thorpe, and Brunet 2015)
Forecasting Competitions
History
Critical statisticians
We should be looking for the true model!
Maybe you do not know how to model with ARIMA…
I suspect it is more likely to depend on the skill of the analyst … these authors are more at home with simple procedures than with Box-Jenkins. (Chatfield)
Makridakis & Hibon (1979)
Our Empirical evidence disagrees
Of course we do
might be useful for Dr Chatfield to read some of the psychological literature quoted in the main paper, and he can then learn a little more about biases
M-Competitions
| Competition | Year | N° Time Series | Insights/Novelty |
|---|---|---|---|
| M1 | 1982 | 1001 |
|
| M2 | 1993 | 29 |
|
| M3 | 2000 | 3003 |
|
| M4 | 2020 | 100,000 |
|
| M5 | 2021 | 42,000 |
|
| M6 | 2022 | 100 |
|
Utility of Competitions
Advantages
- Empirical evidence
- Benchmarking
- Methodological development and cumulative science (Fildes and Ord 2004)
Issues
- arbitrary conditions
- vibe-based analysis of results (Koning et al. 2005)
- Questionable generalizability (Fildes and Ord 2004)
Model Selection, Uncertainty & Combination
Model selection
The beauty of simple models:
Simple models often outperform more complex ones, even on larger data sets (M1, M3)
Simple models are important benchmarks, even in more complex settings
Selecting a single model:
- ignores uncertainty about this selection (cite Kaplan)
- tends to perform poorly in forecasting competitions
Move away from model selection to model combination
Model combination
Test stuff
this
Ensemble modeling
Met Office UK & Bauer, Thorpe, and Brunet (2015)
Relevance for psychology
- Model uncertainty is often neglected, both in inferential and predictive modeling (Kaplan 2021)
- Current predictive literature often seems to neglect model uncertainty
- Theoretically linked to the complex systems literature
Probabilistic Forecasting
Probabilistic forecasting methods
What is probabilistic forecasting?
- what is even meant here?
- show plot by gneiting in the appendix
- show different forms
Moving towards probabilistic forecasting
Weather forecasting used probabilistic forecasting surprisingly early:
The probability of rain was much smaller than at other times (Dalton, 1793)
Popularized by Epstein 1969 Stochastic dynamic prediction
Communicating uncertainty
https://www.weather.gov/mrx/probeducation
Relevance for psychology
Practically linked to decision theory, e.g., for JITAIs
Highly relevant in healthcare settings in general
Mixed Models
Extending mixed models
Flexible Mixed Model
From
\[ y = X\beta + Z\upsilon + \epsilon \]
to
\[y = ml_{fixed}(X)+Z\upsilon + \epsilon\]
- not ‘forecasting’ literature, but ML more broadly
- use of random effects in machine learning has gained attention
Relevance for Psychology
- Improving on what we already have
- Current papers: often rather ad-hoc combinations of forecasts
- Lots of flexibility
- For example:
- lasso with random effects
- random forest/trees with random effects
- boosting with random effects
- …
Summary
Issues
Methodology
- Reproducibility Issues (Boylan et al. 2015)
- ‘Winner takes it all’? (see Strobl and Leisch (2022))
Transfer to Psychology
- Data structure often not comparable (Makridakis, Hyndman, and Petropoulos 2020)
- Much of it is known, but not used
Takeaways
- Forecasting competitions can lead to methodological improvement
- Model uncertainty and model combination are integral parts of forecasting
- Probabilistic forecasting is important for decision making
- Combining predictions is challenging